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CLAIMS 

1. In a system for editing an audiovisual work, a method for producing a retiming effect 
on a clip of synchronized audio data and video data to produce a retimed clip of 
synchronized audio and video data in the audiovisual work, comprising: 

associating a definition of a retiming function for a rampable retiming effect that 
maps output times to input times with the clip of synchronized audio data and video data; 

processing the synchronized audio data and video data according to the retiming 
function to produce the retimed clip, wherein processing comprises: 

for each output time for an audio sample, 

determining a corresponding input time from the output time and the 

retiming function; and 

computing an output audio sample at the output time based on at least the 

audio data in the neighborhood of the corresponding input time using a 

resampling function; and 

for each output time for a video sample, 

determining a corresponding input time from the output time and the 
retiming function, such that input times determined for output times for video 
samples correspond to input times determined for the same output times for audio 
samples; and 

computing an output video sample at the output time based on at least the 
video data in the neighborhood of the corresponding input time using a 
resampling function. 

2. The method of claim 1, wherein the retiming function is defined as a speed curve, and 
wherein the speed curve is integrated to determine the input times from output times for 
both video data and audio data, wherein a step size used to compute an integral of the 
speed curve is less than or equal to a step size corresponding to a sampling rate of the 
audio data. 
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3. The method of claim 1, wherein determining the output audio sample uses a time- 
scaling function. 

4. The method of claim 3 ? wherein determining the output video sample uses motion 
based interpolation. 

5. The method of claim 3, wherein determining the output video sample uses blended 
frames. 

6. The method of claim 1, wherein determining the output audio sample uses a 
resampling function with pitch shifting. 

7. The method of claim 6, wherein determining the output video sample uses motion 
based interpolation. 

8. The method of claim 6, wherein determining the output video sample uses blended 
frames. 

9. The method of claim 1, wherein the neighborhood of the corresponding input time is a 
plurality of samples from points in time surrounding the input time. 

10. The method of claim 9, wherein the audio resampling function generates an output 
audio sample from a plurality of input audio samples at different points in time by 
combining information from the plurality of input audio samples. 

11. The method of claim 9, wherein the audio resampling function generates an output 
audio sample from a plurality of input audio samples at different points in time by 
combining information from the plurality of input audio samples. 
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12. The method of claim 1, wherein an input time determined for any output time for a 
video sample is identical to an input time determined for the same output time for an 
audio sample. 

13. The method of claim 1, wherein an input time determined for each output time for 
video samples is offset from an input time determined for the same output time for audio 
samples. 

14. The method of claim 1, wherein the retiming function comprises a mapping of a 
plurality of video events and a corresponding plurality of audio events to a corresponding 
plurality of output times. 

15. The method of claim 9, further comprising computing a position curve for audio 
from the mapping, and wherein determining an input time from an output time for an 
audio sample uses the position curve. 

16. The method of claim 10, further comprising computing a position curve for video 
from the mapping, and wherein determining an input time from an output time for a video 
sample uses the position curve. 

17. The method of claim 9, further comprising computing a position curve for video 
from the mapping, and wherein determining an input time from an output time for a video 
sample uses the position curve. 

18. The method of claim 9, further comprising receiving an indication of the mapping 
by: 

presenting a graphical user interface including a video track, an audio track and an 
output track; 

receiving an indication of a video event on the video track through an input 

device; 
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receiving an indication of an audio event on the audio track through an input 

device; 

receiving an indication of an output time on the output track through the input 
device; and 

maintaining information indicating a correspondence between the indicated video 
event, the indicated audio event and the indicated output time. 

19. The method of claim 18, further comprising computing a position curve for audio 
from the mapping, and wherein determining an input time from an output time for an 
audio sample uses the position curve. 

20. The method of claim 19, further comprising computing a position curve for video 
from the mapping, and wherein determining an input time from an output time for a video 
sample uses the position curve. 

21. The method of claim 18, further comprising computing a position curve for video 
from the mapping, and wherein determining an input time from an output time for a video 
sample uses the position curve. 

22. A computer program product, comprising: 

a computer readable medium; 

computer program instructions stored in the computer readable medium that, 
when executed by a computer, instruct the computer to perform a method for producing a 
retiming effect on a clip of synchronized audio data and video data to produce a retimed 
clip of synchronized audio and video data in the audiovisual work, comprising: 

associating a definition of a retiming function for a rampable retiming effect that 
maps output times to input times with the clip of synchronized audio data and video data; 

processing the synchronized audio data and video data according to the retiming 
function to produce the retimed clip, wherein processing comprises: 

for each output time for an audio sample, 
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determining a corresponding input time from the output time and the 
retiming function; and 

computing an output audio sample at the output time based on at least the 
audio data in the neighborhood of the corresponding input time using a 
resampling function; and 

for each output time for a video sample, 

determining a corresponding input time from the output time and the 
retiming function, such that input times determined for output times for video 
samples correspond to input times determined for the same output times for audio 
samples; and 

computing an output video sample at the output time based on at least the 
video data in the neighborhood of the corresponding input time using a 
resampling function. 

23. An editing system for editing an audiovisual work and for producing a retiming 
effect on a clip of synchronized audio data and video data to produce a retimed clip of 
synchronized audio and video data in the audiovisual work, comprising: 

means for relating a definition of a retiming function for a rampable retiming 
effect that maps output times to input times with the clip of synchronized audio data and 
video data; 

means for processing the synchronized audio data and video data according to the 
retiming function to produce the retimed clip, comprising: 

means for computing, for each output time for an audio sample, a 
corresponding input time from the output time and the retiming function; 

means for computing an output audio sample at the output time based on 
at least the audio data in the neighborhood of the corresponding input time using a 
resampling function; 

means for computing, for each output time for a video sample, a 
corresponding input time from the output time and the retiming function, such 
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that input times determined for output times for video samples correspond to input 
times determined for the same output times for audio samples; and 

means for computing an output video sample at the output time based on 
at least the video data in the neighborhood of the corresponding input time using a 
resampling function. 

24. An editing system for editing an audiovisual work and for producing a retiming 
effect on a clip of synchronized audio data and video data to produce a retimed clip of 
synchronized audio and video data in the audiovisual work, comprising: 

an editing interface allowing a user to associate a definition of a retiming function 
for a rampable retiming effect that maps output times to input times with the clip of 
synchronized audio data and video data; 

an audio retiming module having an input for receiving the definition of the 
retiming function and an input for receiving the audio data, and an output providing 
retimed audio data such that, for each output time for an audio sample, an output audio 
sample is computed using a resampling function based on at least the audio data in a 
neighborhood of a corresponding input time according to the retiming function; and 

a video retiming module having an input for receiving the definition of the 
retiming function and an input for receiving the video data, and an output providing 
retimed video data such that, for each output time for a video sample, an output video 
sample is computed using a resampling function based on at least the video data in a 
neighborhood of a corresponding input time according to the retiming function, wherein 
input times determined for output times for video samples correspond to input times 
determined for the same output times for audio samples. 

25. A digital entertainment product, comprising: 

a computer readable medium; 

information stored on the computer readable medium that, when read by a 
computer, indicates to the computer a retimed clip of synchronized audio data and video 
data, produced according to a process for retiming a clip of synchronized audio data and 
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video data to produce a retimed clip of synchronized audio and video data in the 
audiovisual work, the process comprising: 

associating a definition of a retiming function for a rampable retiming effect that 
maps output times to input times with the clip of synchronized audio data and video data; 

processing the synchronized audio data and video data according to the retiming 
function to produce the retimed clip, wherein processing comprises: 

for each output time for an audio sample, 

determining a corresponding input time from the output time and the 

retiming function; and 

computing an output audio sample at the output time based on at least the 

audio data in the neighborhood of the corresponding input time using a 

resampling function; and 

for each output time for a video sample, 

determining a corresponding input time from the output time and the 
retiming function, such that input times determined for output times for video 
samples correspond to input times determined for the same output times for audio 
samples; and 

computing an output video sample at the output time based on at least the 
video data in the neighborhood of the corresponding input time using a 
resampling function. 

26. In a system for editing an audiovisual work including a sequence of a plurality of 
clips of audiovisual data, a method for producing a retiming effect on a clip of 
synchronized audio data and video data to produce a retimed clip of synchronized audio 
and video data in the audiovisual work, comprising: 

associating a definition of a retiming function for a rampable retiming effect that 
maps output times to input times with the clip of synchronized audio data and video data; 

processing the video data of the clip according to the retiming function to produce 
a retimed video clip, wherein processing comprises: 
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for each output time for a video sample, 

determining a corresponding input time from the output time and the 
retiming function; and 

computing an output video sample at the output time based on at least the 
video data in the neighborhood of the corresponding input time using a 
resampling function; 

placing the retimed video clip in the audiovisual work; and 
exporting the audiovisual work to an audio editing system, including the 
definition of the retiming function, for processing the audio data of the clip according to 
the retiming function, such that an input time determined for each output time for video 
samples corresponds to an input time determined for the same output time for audio 
samples. 

27. In a system for editing an audiovisual work including a sequence of a plurality of 
clips of audiovisual data, a method for producing a retiming effect on a clip of 
synchronized audio data and video data to produce a retimed clip of synchronized audio 
and video data in the audiovisual work, comprising: 

receiving a definition of a retiming function for a rampable retiming effect that 
maps output times to input times associated with the clip of synchronized audio data and 
video data; 

receiving the audio data; 

receiving the audiovisual work including a retimed video clip processed according 
to the retiming function; 

processing the audio data according to the retiming function to produce a retimed 
audio clip, wherein processing comprises: 

for each output time for an audio sample, 

determining a corresponding input time from the output time and the 
retiming function, such that an input time determined for each output time for 
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video samples corresponds to an input time determined for the same output time 
for audio samples; and 

computing an output audio sample at the output time based on at least the 
audio data in the neighborhood of the corresponding input time using a 
resampling function; and 

synchronizing the retimed audio clip with the retimed video clip in the 
audiovisual work. 

28. In a system for editing an audiovisual work including a sequence of a plurality of 
clips of audiovisual data, a method for producing a retiming effect on a clip of 
synchronized audio data and video data to produce a retimed clip of synchronized audio 
and video data in the audiovisual work, comprising: 

receiving a definition of a retiming function for a rampable retiming effect that 
maps output times to input times associated with the clip of synchronized audio data and 
video data; 

receiving the video data; 

receiving the audiovisual work including a retimed audio clip processed according 
to the retiming function; 

processing the video data according to the retiming function to produce a retimed 
video clip, wherein processing comprises: 

for each output time for a video sample, 

determining a corresponding input time from the output time and the 
retiming function, such that an input time determined for each output time for 
video samples corresponds to an input time determined for the same output time 
for audio samples; and 

computing an output video sample at the output time based on at least the 
video data in the neighborhood of the corresponding input time using a 
resampling function; and 
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synchronizing the retimed video clip with the retimed audio clip in the audiovisual work. 

29. In a system for editing an audiovisual work, a method for defining a retiming effect 
applied to audio data and video data to produce a retimed clip of synchronized audio and 
video data in the audiovisual work, comprising: 

associating a mapping of a plurality of video events in the video data and a 
corresponding plurality of audio events in the audio data to a corresponding plurality of 
output times in the retimed clip; 

processing the audio data according to the retiming function to produce retimed 
audio data; 

processing the video data according to the retiming function to produce retimed 
video data; 

placing the retimed audio data and retimed video data in the audiovisual work as 
the retimed clip of synchronized audio and video data. 

30. The method of claim 29, wherein processing the audio data comprises 

computing a position curve for audio from the mapping; 
for each output time for an audio sample, 

determining a corresponding input time from the output time using the 
position curve; and 

computing an output audio sample at the output time based on at least the 
audio data in the neighborhood of the corresponding input time using a 
resampling function. 

31. The method of claim 30, wherein processing the video data comprises 

computing a position curve for video from the mapping; 
for each output time for a video sample, 

determining a corresponding input time from the output time using the 
position curve; and 
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computing an output video sample at the output time based on at least the 
video data in the neighborhood of the corresponding input time using a 
resampling function. 

32. The method of claim 29, wherein processing the video data comprises 

computing a position curve for video from the mapping; 
for each output time for a video sample, 

determining a corresponding input time from the output time using the 
position curve; and 

computing an output video sample at the output time based on at least the 
video data in the neighborhood of the corresponding input time using a 
resampling function. 

33. The method of claim 29, further comprising receiving an indication of the mapping 
by: 

presenting a graphical user interface including a video track, an audio track and an 
output track; 

receiving an indication of a video event on the video track through an input 

device; 

receiving an indication of an audio event on the audio track through an input 

device; 

receiving an indication of an output time on the output track through the input 
device; and 

maintaining information indicating a correspondence between the indicated video 
event, the indicated audio event and the indicated output time. 

34. A computer program product, comprising: 

a computer readable medium; 

computer program instructions stored on the computer readable medium that, 
when executed by a computer, instructs the computer to perform a method for defining a 
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retiming effect applied to audio data and video data to produce a retimed clip of 
synchronized audio and video data in the audiovisual work, comprising: 

associating a mapping of a plurality of video events in the video data and a 
corresponding plurality of audio events in the audio data to a corresponding plurality of 
output times in the retimed clip; 

processing the audio data according to the retiming function to produce retimed 
audio data; 

processing the video data according to the retiming function to produce retimed 
video data; 

placing the retimed audio data and retimed video data in the audiovisual work as 
the retimed clip of synchronized audio and video data. 

35. An editing system for defining a retiming effect applied to audio data and video data 
to produce a retimed clip of synchronized audio and video data in the audiovisual work, 
comprising: 

means for mapping a plurality of video events in the video data and a 
corresponding plurality of audio events in the audio data to a corresponding plurality of 
output times in the retimed clip; 

means for processing the audio data according to the retiming function to produce 
retimed audio data; 

means for processing the video data according to the retiming function to produce 
retimed video data; and 

means for inserting the retimed audio data and retimed video data in the 
audiovisual work as the retimed clip of synchronized audio and video data. 

36. A method for applying a rampable retiming effect to synchronized streams of 
temporal media data, include a first stream of a first media type and a second stream of a 
second media type different from the first media type, comprising: 

associating a retiming function for the rampable retiming effect with the 
synchronized streams, wherein the retiming function maps output times to input times; 
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processing the synchronized streams according to the retiming function, wherein 
processing comprises: 

for each output time for the first stream, 

determining a corresponding input time from the output time and 
the retiming function; and 

computing an output sample for the first stream at the output time 
from at least data in a neighborhood of the corresponding input time, using 
a resampling function for the first media type; and 
for each output time for the second stream, 

determining a corresponding input time from the output time and 
the retiming function, such that an input time determined for an output 
time for a sample of the second stream corresponds to an input time 
determined for the same output times for a sample of the first stream; and 

computing an output sample for the second stream at the output 
time from at least data in a neighborhood of the corresponding input time, 
using a resampling function for the second media type. 

37. A method for applying a rampable retiming effect to synchronized streams of 
temporal media data, wherein the streams include a first stream of a first media type and 
a second stream of a second media type, comprising: 

associating a retiming function for the rampable retiming effect with the 
synchronized streams, wherein the retiming function defines a mapping of output times to 
input times, wherein the mapping has a corresponding temporal resolution of greater than 
or equal to a temporal resolution of both of the synchronized streams; 

processing the synchronized streams according to the retiming function, wherein 
processing comprises: 

for each output time for the first stream, 

determining a corresponding input time from the output time and 
the mapping defined by the retiming function; and 

computing an output sample for the first stream at the output time 
from at least data in a neighborhood of the corresponding input time, using 
a resampling function for the first media type; and 
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for each output time for the second stream, 

determining a corresponding input time from the output time and 
the mapping defined by the retiming function; and 

computing an output sample for the second stream at the output 
time from at least data in a neighborhood of the corresponding input time, 
using a resampling function for the second media type. 

38. A method for applying a rampable retiming effect to temporal media data 
synchronized with metadata, comprising: 

associating a retiming function for the rampable retiming effect with the temporal 
media data, wherein the retiming function maps output times to input times; 

processing the temporal media data according to the retiming function, wherein 
processing comprises, for each output time, 

determining a corresponding input time from the output time and the 

retiming function; and 

computing an output sample of the temporal media data at the output time 

from at least temporal media data in a neighborhood of the corresponding input 

time, using a resampling function for the temporal media data; and 
processing the metadata to synchronize the metadata with the retimed temporal media 
data. 



